Quickstart - Customer Churn Full Suite Model Documentation

This interactive notebook will guide you through documenting a model using the ValidMind Developer framework. We will use sample datasets provided by the library and train a simple classification model.

For this simple demonstration, we will use the following bank customer churn dataset from Kaggle: https://www.kaggle.com/code/kmalit/bank-customer-churn-prediction/data.

We will train a sample model and demonstrate the following documentation functionalities:

• Initializing the ValidMind Developer Framework
• Using a sample datasets provided by the library to train a simple classification model
• Running a test suite to quickly generate document about the data and model

Install ValidMind Developer Framework

%pip install --upgrade validmind

Requirement already satisfied: validmind in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (1.19.2)
Requirement already satisfied: shap<0.42.0,>=0.41.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.41.0)
Requirement already satisfied: tqdm<5.0.0,>=4.64.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (4.64.1)
Requirement already satisfied: rouge<2.0.0,>=1.0.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.0.1)
Requirement already satisfied: pandas==1.5.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.5.3)
Requirement already satisfied: statsmodels<0.14.0,>=0.13.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.13.5)
Requirement already satisfied: plotly<6.0.0,>=5.14.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (5.17.0)
Requirement already satisfied: click<9.0.0,>=8.0.4 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (8.1.3)
Requirement already satisfied: xgboost<2.0.0,>=1.5.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.7.4)
Requirement already satisfied: pypmml<0.10.0,>=0.9.17 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.9.17)
Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (3.8.1)
Requirement already satisfied: plotly-express<0.5.0,>=0.4.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.4.1)
Requirement already satisfied: textstat<0.8.0,>=0.7.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.7.3)
Requirement already satisfied: aiohttp[speedups]<4.0.0,>=3.8.4 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (3.8.5)
Requirement already satisfied: langdetect<2.0.0,>=1.0.9 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.0.9)
Requirement already satisfied: numpy==1.22.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.22.3)
Requirement already satisfied: seaborn<0.12.0,>=0.11.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.11.2)
Requirement already satisfied: pandas-profiling<4.0.0,>=3.6.6 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (3.6.6)
Requirement already satisfied: dython<0.8.0,>=0.7.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.7.2)
Requirement already satisfied: sentencepiece<0.2.0,>=0.1.99 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.1.99)
Requirement already satisfied: ipython==7.34.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (7.34.0)
Requirement already satisfied: ipywidgets<9.0.0,>=8.0.6 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (8.1.1)
Requirement already satisfied: pydantic<2.0.0,>=1.9.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.10.5)
Requirement already satisfied: tabulate<0.9.0,>=0.8.9 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.8.10)
Requirement already satisfied: python-dotenv<0.21.0,>=0.20.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.20.0)
Requirement already satisfied: scorecardpy<0.2.0.0,>=0.1.9.6 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (0.1.9.7)
Requirement already satisfied: sentry-sdk<2.0.0,>=1.24.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.31.0)
Requirement already satisfied: catboost<2.0,>=1.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.2.1)
Requirement already satisfied: pdoc<14.0.0,>=13.1.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (13.1.1)
Requirement already satisfied: arch<6.0.0,>=5.4.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (5.6.0)
Requirement already satisfied: markdown<4.0.0,>=3.4.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (3.4.4)
Requirement already satisfied: wget<4.0,>=3.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (3.2)
Requirement already satisfied: requests<3.0.0,>=2.27.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (2.28.2)
Requirement already satisfied: scikit-learn<2.0.0,>=1.0.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from validmind) (1.2.1)
Requirement already satisfied: backcall in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (0.2.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (3.0.38)
Requirement already satisfied: pickleshare in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (0.7.5)
Requirement already satisfied: matplotlib-inline in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (0.1.6)
Requirement already satisfied: pexpect>4.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (4.8.0)
Requirement already satisfied: appnope in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (0.1.3)
Requirement already satisfied: traitlets>=4.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (5.9.0)
Requirement already satisfied: pygments in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (2.14.0)
Requirement already satisfied: jedi>=0.16 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (0.18.2)
Requirement already satisfied: setuptools>=18.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (67.4.0)
Requirement already satisfied: decorator in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipython==7.34.0->validmind) (5.1.1)
Requirement already satisfied: python-dateutil>=2.8.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pandas==1.5.3->validmind) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pandas==1.5.3->validmind) (2022.7.1)
Requirement already satisfied: aiosignal>=1.1.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (1.3.1)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (3.0.1)
Requirement already satisfied: frozenlist>=1.1.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (1.4.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (4.0.3)
Requirement already satisfied: multidict<7.0,>=4.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (6.0.4)
Requirement already satisfied: yarl<2.0,>=1.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (1.9.2)
Requirement already satisfied: attrs>=17.3.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (22.2.0)
Requirement already satisfied: Brotli in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (1.1.0)
Requirement already satisfied: cchardet in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (2.1.7)
Requirement already satisfied: aiodns in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (3.0.0)
Requirement already satisfied: scipy>=1.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from arch<6.0.0,>=5.4.0->validmind) (1.9.3)
Requirement already satisfied: property-cached>=1.6.4 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from arch<6.0.0,>=5.4.0->validmind) (1.6.4)
Requirement already satisfied: matplotlib in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from catboost<2.0,>=1.2->validmind) (3.6.3)
Requirement already satisfied: six in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from catboost<2.0,>=1.2->validmind) (1.16.0)
Requirement already satisfied: graphviz in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from catboost<2.0,>=1.2->validmind) (0.20.1)
Requirement already satisfied: scikit-plot>=0.3.7 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from dython<0.8.0,>=0.7.1->validmind) (0.3.7)
Requirement already satisfied: psutil>=5.9.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from dython<0.8.0,>=0.7.1->validmind) (5.9.4)
Requirement already satisfied: jupyterlab-widgets~=3.0.9 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipywidgets<9.0.0,>=8.0.6->validmind) (3.0.9)
Requirement already satisfied: comm>=0.1.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipywidgets<9.0.0,>=8.0.6->validmind) (0.1.4)
Requirement already satisfied: widgetsnbextension~=4.0.9 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ipywidgets<9.0.0,>=8.0.6->validmind) (4.0.9)
Requirement already satisfied: importlib-metadata>=4.4 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from markdown<4.0.0,>=3.4.3->validmind) (6.0.0)
Requirement already satisfied: joblib in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from nltk<4.0.0,>=3.8.1->validmind) (1.2.0)
Requirement already satisfied: regex>=2021.8.3 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from nltk<4.0.0,>=3.8.1->validmind) (2023.8.8)
Requirement already satisfied: ydata-profiling in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pandas-profiling<4.0.0,>=3.6.6->validmind) (4.0.0)
Requirement already satisfied: MarkupSafe in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pdoc<14.0.0,>=13.1.1->validmind) (2.1.2)
Requirement already satisfied: Jinja2>=2.11.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pdoc<14.0.0,>=13.1.1->validmind) (3.1.2)
Requirement already satisfied: tenacity>=6.2.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from plotly<6.0.0,>=5.14.1->validmind) (8.2.3)
Requirement already satisfied: packaging in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from plotly<6.0.0,>=5.14.1->validmind) (23.0)
Requirement already satisfied: patsy>=0.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from plotly-express<0.5.0,>=0.4.1->validmind) (0.5.3)
Requirement already satisfied: typing-extensions>=4.2.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pydantic<2.0.0,>=1.9.1->validmind) (4.5.0)
Requirement already satisfied: py4j>=0.10.7 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pypmml<0.10.0,>=0.9.17->validmind) (0.10.9.7)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from requests<3.0.0,>=2.27.1->validmind) (1.26.14)
Requirement already satisfied: idna<4,>=2.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from requests<3.0.0,>=2.27.1->validmind) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from requests<3.0.0,>=2.27.1->validmind) (2022.12.7)
Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from scikit-learn<2.0.0,>=1.0.2->validmind) (3.1.0)
Requirement already satisfied: numba in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from shap<0.42.0,>=0.41.0->validmind) (0.56.4)
Requirement already satisfied: cloudpickle in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from shap<0.42.0,>=0.41.0->validmind) (2.2.1)
Requirement already satisfied: slicer==0.0.7 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from shap<0.42.0,>=0.41.0->validmind) (0.0.7)
Requirement already satisfied: pyphen in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from textstat<0.8.0,>=0.7.3->validmind) (0.14.0)
Requirement already satisfied: zipp>=0.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown<4.0.0,>=3.4.3->validmind) (3.15.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from jedi>=0.16->ipython==7.34.0->validmind) (0.8.3)
Requirement already satisfied: pillow>=6.2.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from matplotlib->catboost<2.0,>=1.2->validmind) (9.4.0)
Requirement already satisfied: cycler>=0.10 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from matplotlib->catboost<2.0,>=1.2->validmind) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from matplotlib->catboost<2.0,>=1.2->validmind) (4.38.0)
Requirement already satisfied: pyparsing>=2.2.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from matplotlib->catboost<2.0,>=1.2->validmind) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from matplotlib->catboost<2.0,>=1.2->validmind) (1.4.4)
Requirement already satisfied: contourpy>=1.0.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from matplotlib->catboost<2.0,>=1.2->validmind) (1.0.7)
Requirement already satisfied: ptyprocess>=0.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pexpect>4.3->ipython==7.34.0->validmind) (0.7.0)
Requirement already satisfied: wcwidth in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython==7.34.0->validmind) (0.2.6)
Requirement already satisfied: pycares>=4.0.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from aiodns->aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (4.3.0)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from numba->shap<0.42.0,>=0.41.0->validmind) (0.39.1)
Requirement already satisfied: PyYAML<6.1,>=5.0.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (6.0)
Requirement already satisfied: typeguard<2.14,>=2.13.2 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (2.13.3)
Requirement already satisfied: multimethod<1.10,>=1.4 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (1.9.1)
Requirement already satisfied: phik<0.13,>=0.11.1 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (0.12.3)
Requirement already satisfied: htmlmin==0.1.12 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (0.1.12)
Requirement already satisfied: visions[type_image_path]==0.7.5 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (0.7.5)
Requirement already satisfied: networkx>=2.4 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from visions[type_image_path]==0.7.5->ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (3.0)
Requirement already satisfied: tangled-up-in-unicode>=0.0.4 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from visions[type_image_path]==0.7.5->ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (0.2.0)
Requirement already satisfied: imagehash in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from visions[type_image_path]==0.7.5->ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (4.3.1)
Requirement already satisfied: cffi>=1.5.0 in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from pycares>=4.0.0->aiodns->aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (1.15.1)
Requirement already satisfied: pycparser in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from cffi>=1.5.0->pycares>=4.0.0->aiodns->aiohttp[speedups]<4.0.0,>=3.8.4->validmind) (2.21)
Requirement already satisfied: PyWavelets in /Users/nrichers/Library/Caches/pypoetry/virtualenvs/validmind-Tie81wAs-py3.9/lib/python3.9/site-packages (from imagehash->visions[type_image_path]==0.7.5->ydata-profiling->pandas-profiling<4.0.0,>=3.6.6->validmind) (1.4.1)

[notice] A new release of pip is available: 23.0 -> 23.2.1
[notice] To update, run: python3 -m pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

Initializing the Python environment

import pandas as pd
import xgboost as xgb

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

%matplotlib inline

Initializing the ValidMind Client Library

Log in to the ValidMind platform with your registered email address, and navigate to the Documentation Projects page.

Creating a new Documentation Project

(Note: if a documentation project has already been created, you can skip this section and head directly “Finding Project API key and secret”)

Clicking on “Create a new project” allows to you to register a new documentation project for our demo model.

Select “Customer Churn model” from the Model drop-down, and “Initial Validation” as Type. Finally, click on “Create Project”.

Finding the project API key and secret

In the “Client Integration” page of the newly created project, you will find the initialization code that allows the client library to associate documentation and tests with the appropriate project. The initialization code configures the following arguments:

• api_host: Location of the ValidMind API.
• api_key: Account API key.
• api_secret: Account Secret key.
• project: The project identifier. The project argument is mandatory since it allows the library to associate all data collected with a specific account project.

The code snippet can be copied and pasted directly in the cell below to initialize the ValidMind Developer Framework when run:

## Replace the code below with the code snippet from your project ## 

import validmind as vm

vm.init(
  api_host = "https://api.dev.vm.validmind.ai/api/v1/tracking",
  api_key = "bc106842f5958db49f389f04d7c01c78",
  api_secret = "b84dafdfdb78ecf21f4f815e269da7e187e9659bf487138afe15c42edcb3e0bb",
  project = "clmkv739v000q19mojndgpqea"
)

2023-09-15 15:03:18,658 - INFO(validmind.api_client): Connected to ValidMind. Project: Nik — Test 2  - Initial Validation (clmkv739v000q19mojndgpqea)

Load the Demo Dataset

For the purpose of this demonstration, we will use a sample dataset provided by the ValidMind library.

# Import the sample dataset from the library
from validmind.datasets.classification import customer_churn as demo_dataset
# You can try a different dataset with: 
#from validmind.datasets.classification import taiwan_credit as demo_dataset

df = demo_dataset.load_data()

Initialize a dataset object for ValidMind

Before running the test plan, we must first initialize a ValidMind dataset object using the init_dataset function from the vm module. This function takes in arguements: dataset which is the dataset that we want to analyze; target_column which is used to identify the target variable; class_labels which is used to identify the labels used for classification model training.

vm_dataset = vm.init_dataset(
    dataset=df,
    target_column=demo_dataset.target_column,
    class_labels=demo_dataset.class_labels
)

2023-09-15 15:03:18,674 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...

Run the Full Data and Model Validation Test Suite

We will need to preprocess the dataset and produce the training, test and validation splits first.

Prepocess the Raw Dataset

For demonstration purposes, we simplified the preprocessing using demo_dataset.preprocess which executes the following operations:

train_df, validation_df, test_df = demo_dataset.preprocess(df)

x_train = train_df.drop(demo_dataset.target_column, axis=1)
y_train = train_df[demo_dataset.target_column]
x_val = validation_df.drop(demo_dataset.target_column, axis=1)
y_val = validation_df[demo_dataset.target_column]

model = xgb.XGBClassifier(early_stopping_rounds=10)
model.set_params(
    eval_metric=["error", "logloss", "auc"],
)
model.fit(
    x_train,
    y_train,
    eval_set=[(x_val, y_val)],
    verbose=False,
)

XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=None, early_stopping_rounds=10,
              enable_categorical=False, eval_metric=['error', 'logloss', 'auc'],
              feature_types=None, gamma=None, gpu_id=None, grow_policy=None,
              importance_type=None, interaction_constraints=None,
              learning_rate=None, max_bin=None, max_cat_threshold=None,
              max_cat_to_onehot=None, max_delta_step=None, max_depth=None,
              max_leaves=None, min_child_weight=None, missing=nan,
              monotone_constraints=None, n_estimators=100, n_jobs=None,
              num_parallel_tree=None, predictor=None, random_state=None, ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

We can now initialize the training and test datasets into dataset objects using vm.init_dataset():

vm_train_ds = vm.init_dataset(
    dataset=train_df,
    type="generic",
    target_column=demo_dataset.target_column
)

vm_test_ds = vm.init_dataset(
    dataset=test_df,
    type="generic",
    target_column=demo_dataset.target_column
)

2023-09-15 15:03:18,779 - INFO(validmind.client): The 'type' argument to init_dataset() argument is deprecated and no longer required.
2023-09-15 15:03:18,779 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...
2023-09-15 15:03:18,810 - INFO(validmind.client): The 'type' argument to init_dataset() argument is deprecated and no longer required.
2023-09-15 15:03:18,810 - INFO(validmind.client): Pandas dataset detected. Initializing VM Dataset instance...

We also initialize a model object using vm.init_model():

vm_model = vm.init_model(
    model,
    train_ds=vm_train_ds,
    test_ds=vm_test_ds,
)

Run the Full Suite

!pip uninstall kaleido -y && pip install -U kaleido

Found existing installation: kaleido 0.2.1
Uninstalling kaleido-0.2.1:
  Successfully uninstalled kaleido-0.2.1
Collecting kaleido
  Using cached kaleido-0.2.1-py2.py3-none-macosx_11_0_arm64.whl (85.8 MB)
Installing collected packages: kaleido
Successfully installed kaleido-0.2.1

We are now ready to run the test suite for binary classifier with tabular datasets. This function will run test plans on the dataset and model objects, and will document the results in the ValidMind UI.

full_suite = vm.run_test_suite(
    "classifier_full_suite",
    dataset=vm_dataset,
    model=vm_model
)

2023-09-15 15:03:30,556 - ERROR(validmind.vm_models.test_plan): Failed to run test 'models_performance_comparison': (MissingRequiredTestContextError) Models 'models' is a required input and must be passed as a keyword argument to the test plan

You can access and review the resulting documentation in the ValidMind UI, in the “Model Development” section of the model documentation.